Skip to content

Conversation

@bmeagherix
Copy link
Contributor

@bmeagherix bmeagherix commented May 28, 2025

Add middleware plumbing for SPDK NVMe-oF backend.

Depending on the kernel setting in nvmet.global.config the configuration will either be applied to the kernel NVMe-oF target, or to a SPDK based target. This PR adds support for the latter.

Overview

If the kernel implementation is not selected, then a user-space daemon is launched. The ix-nvmf service is a thin wrapper around the /opt/spdk/usr/bin/nvmf_tgt executable that calls into middleware to perform some necessary setup/teardown. Before the executable is launched, some DPDK setup is performed, and after it has been launched the configuration is "applied" to it.

A python library is delivered as part of SPDK, and is called directly from middleware to configure the daemon. So, it will "create" ports, subsystems, namespaces, bdevs, etc within the daemon.

Tests

Existing NVMe-oF tests are now called both for kernel and for SPDK implementations, using parametrized fixtures. Some minor test fixup was required, most especially because SPDK is more "generous" in referral creation.

Missing

The following items have not been addressed in this PR:

  1. Optionally unbinding NICs from the kernel, and using vfio-pci instead (i.e. dedicated NICs).
  2. Gracefully handling less capable CPUs

Following PR review/address review/PR approve did another CI run here. Passes all NVMe-oF tests.

@bmeagherix bmeagherix self-assigned this May 28, 2025
@bmeagherix bmeagherix added the WIP label May 28, 2025
@bugclerk bugclerk changed the title NAS-135917 NAS-135917 / 25.10 / NAS-135917 May 28, 2025
@bugclerk
Copy link
Contributor

@bmeagherix bmeagherix changed the title NAS-135917 / 25.10 / NAS-135917 NAS-135917 / 25.10 / SPDK backend for NVMe-oF May 29, 2025
@bmeagherix bmeagherix force-pushed the NAS-135917 branch 3 times, most recently from 5b0ee9d to 008511a Compare June 4, 2025 14:46
@bmeagherix bmeagherix force-pushed the NAS-135917 branch 3 times, most recently from d29699e to 74337d2 Compare June 9, 2025 18:10
@bmeagherix bmeagherix force-pushed the NAS-135917 branch 2 times, most recently from d2afe48 to bc32349 Compare June 13, 2025 14:31
@bmeagherix bmeagherix force-pushed the NAS-135917 branch 2 times, most recently from 6507884 to a7dbb72 Compare June 25, 2025 14:33
@bmeagherix bmeagherix force-pushed the NAS-135917 branch 2 times, most recently from 13b5239 to 9c2692c Compare July 16, 2025 19:44
@bmeagherix bmeagherix changed the title NAS-135917 / 25.10 / SPDK backend for NVMe-oF NAS-135917 / 26.04 / SPDK backend for NVMe-oF Oct 13, 2025
@bmeagherix bmeagherix removed the WIP label Oct 17, 2025
@bmeagherix bmeagherix requested a review from a team October 17, 2025 13:15
@bmeagherix bmeagherix requested a review from yocalebo October 17, 2025 15:09
@bmeagherix bmeagherix merged commit 242d669 into master Oct 17, 2025
1 of 2 checks passed
@bmeagherix bmeagherix deleted the NAS-135917 branch October 17, 2025 17:30
@bugclerk
Copy link
Contributor

JIRA ticket https://ixsystems.atlassian.net/browse/NAS-135917 is targeted to the following versions which have not received their corresponding PRs: 26.04

@bugclerk
Copy link
Contributor

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.

@truenas truenas locked as resolved and limited conversation to collaborators Oct 17, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants